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Needle in a Haystack? Seeking Causal Evidence about Using STEM Experts to Improve 

Student Outcomes 

Background / Context: 

The vast majority of American students are neither prepared nor sufficiently engaged to 
become STEM-literate citizens or innovative science, technology, engineering, and mathematics 
(STEM) professionals (National Research Council, 2007; Carnegie Corporation and the Institute 
for Advanced Study, 2009; National Science Board [NSB], 2010; President’s Council of 
Advisors on Science and Technology [PCAST], 2010). Evidence from the 2012 Program for 
International Student Assessment (PISA) international assessment places the U.S. in the bottom 
third in science (20 th of 34 OEPD nations), and bottom fourth in mathematics achievement (27 th 
of 34) (PISA, 2012). 1 Scores from the National Assessment of Educational Progress (NAEP) 
indicate that less than one-third of U.S. eighth graders show proficiency in mathematics and 
science, with African-American, Hispanic, and Native American students consistently 
underperforming compared to their white peers (National Center for Education Statistics 
[NCES], 2013), and less than 9 percent of U.S. 15-year olds were top performers (level 5 or 6) as 
measured by PISA (PISA, 2013). These results clearly indicate that American students are ill- 
prepared for advanced scientific training or more rigorous STEM courses necessary to pursue 
post-secondary education and/or careers in the STEM fields. 

Moreover, the challenge of developing STEM-literate citizens and building the STEM 
professional pipeline is multifaceted, and represents more than a lack of academic preparation or 
achievement. Evidence about students’ interest in science — which predicts students’ pursuit of 
science-related careers — is a critical part of the puzzle, as too many high school students report 
that they dismiss STEM career possibilities because they neither know people who work in 
STEM areas nor understand what such people do (Tai et al., 2006; Lemelson, 2010). The interest 
gap is particularly severe among girls and minorities; research indicates that members of these 
groups are far less likely to pursue post-secondary coursework or graduate with STEM degrees 
than their white and/or male counterparts (Higher Education Research Institute Research Brief, 
2010). Increasing students’ interest in STEM is an essential step in increasing their subsequent 
pursuit of STEM education and careers as well as the general competency expected of U.S. 
citizens in the 21 st century workforce (NSB, 2010, 2014). 

In today’ s technological and global society, STEM disciplines are viewed as fundamental to 
the nation’s economic growth and prosperity. Employment opportunities in STEM fields have 
increased at a faster rate than in non-STEM fields (Government Accounting Office [GAO], 
2006). Additionally, many professions — once perceived not to require STEM skills — are 
increasingly requiring scientific and technological proficiency (PCAST, 2010). As American 
students continue to underperform in math and science compared to their peers internationally, 
concerns have arisen about America’s economic and technological competitiveness. Research 
also indicates that a large share of U.S. science degrees are awarded to people born abroad 
(Borjas, 2005), and America’s dependence on foreign-born and foreign-trained scientists is on 
the rise (Xie & Achen, 2009). Such trends have sparked political interest in uncovering means to 
more effectively prepare and engage students in these fields. In 2013, a report from the White 
House Office of Science and Technology Policy called for “a concerted and inclusive effort to 


1 Evidence from the most recent TIMMS assessments suggest somewhat better results for 4 th and 8 th grade 

students; American 4 th graders’ scores were in the top third, and 8 th graders’ scores in the top half for both math 
and science, respectively (see http://nces.ed. gov/timss/results 1 1 . asp ). 


SREE Spring 2015 Conference Abstract Template 


1 


ensure that the STEM workforce is equipped with the skills and training needed to excel in these 
fields” to sustain a capable and flexible workforce that can compete in the global marketplace 
(National Science and Technology Council, 2013). 

Policy interest extends to the role of professionals in STEM education of students. For 
example, in April 2013, President Obama announced that the Corporation for National and 
Community Service (CNCS) would launch a STEM AmeriCorps initiative, to build student 
interest in science and engineering, by mobilizing AmeriCorps national service members in 
nonprofit organizations to work with STEM professionals to inspire young people to excel in 
STEM education (CNCS, 2013), and recently CNCS announced an expansion of this effort 
(CNCS, 2014). 2013 also saw the launch of US2020, an ambitious national partnership 
organization focused on preparing students for STEM-related careers by matching girls, 
underrepresented minorities, and low-income children with STEM mentors.' By providing 
students with a broader range of exposure to STEM content, and more relevant content, from 
more diverse adults than their teachers alone — including young adults in STEM-related 
careers — US2020 hopes increasing numbers of students will be compelled to learn more about 
STEM subjects, and ultimately, pursue additional STEM education and career possibilities. 

As evident by the diverse range of existing initiatives aimed at increasing interest and 
improving academic performance in STEM, the challenge our nation faces is complex, and a 
resolution requires a multipronged approach at both the local and national levels. Change 
requires initiatives that can have multiple positive impacts, both promoting student interest and 
academic achievement, as well as engendering a greater societal change that will help develop 
and sustain a STEM-literate citizenry. Accelerating greater interest, competence and 
achievement in STEM fields requires thoughtful analysis of existing research to parse out what 
we already know about effective strategies for such acceleration, and to highlight where 
additional research may be warranted. In fact, within the past several months alone, the National 
Academies of Science issued a report about integration of STEM into K-12 education, and 
convened an invited workshop summarizing best practices in informal science, and the GAO 
issued a report on connections between STEM education and the workforce. 

Research on youth development may offer important and applicable lessons about how to 
most effectively engage students. The youth development literature has found, for example, that 
hands-on project learning (often called inquiry learning) and peer-to-peer interactions have 
positive impacts on metacognitive development, academic outcomes, and student motivation 
(Flick, 1993; Haury & Rillero, 1994). Hands-on learning often enables students to work together 
in groups and, in turn, develop social skills. Such opportunities for students to talk through 
course materials with peers have been found to help students learn in different ways and retain 
information more effectively (Johnson & Johnson, 1986). Additionally, research suggests that 
hands-on learning positively influences students’ attitudes about the content they are learning 
(e.g., Gerstner & Bogner, 2010; Randier & Hulde, 2007). 


2 See http://us2020.org/stem-mentoring/ . 

3 See http://sites.nationalacademies.org/DBASSE/BOSE/DBASSE 086989 . and 
http://www.nap.edu/catalog.php7record id=18612 for information on the National Academies efforts, and 
http://www.gao.gov/assets/670/663079.pdf for the GAO’s report. 
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The research also suggests that opportunities for adolescents to have meaningful 
engagements and supportive relationships with adults can influence a range of outcomes, 
including educational performance, mental health, and problem behavior (DuBois & Silverthorn, 
2005; Eby et al., 2008; Rhodes & DuBois, 2008). The presence of positive adult role models, 
particularly in the form of mentor relationships, has been shown to have benefits for adolescents 
across academic and socio-emotional domains (Coleman, 1988; Jekielek, Moore, Hair, & 
Scarupa, 2002; Karcher, 2008; Werner & Smith, 1982). A meta-analysis of research about 
mentoring found that mentoring programs particularly benefit at-risk youth (DuBois, Holloway, 
Valentine, & Cooper, 2002), and an impact study of the Big Brothers/Big Sisters program found 
that mentored students showed greater scholastic competence, higher attendance rates and grades 
than those without mentors (Tierney, Grossman, & Resch, 2000). 

The research base about youth development provides useful insights for thinking about how 
best practices for adolescent engagement could apply to programs designed to spur educational 
and ultimately career interests in STEM. One common element across many current STEM 
initiatives is expanding the number and types of adults with whom students interact about STEM 
careers and learning. More specifically, many programs have sought to incorporate mentoring 
relationships, whether the mentors serve as role models, coaches, informal or formal educators, 
or as representatives of individuals who work in diverse STEM content areas, to support 
engagement with the STEM fields. Given that mentoring relationships have been shown to 
positively influence academic and developmental outcomes for students, there is hope that 
STEM-specific mentoring programs have the potential to provide some of the same benefits with 
supplemental content exposure and support. 

Purpose / Objective / Research Question / Focus of Study: 

For this project, we focus specifically on projects that engage STEM-trained adults to work 
directly with students and identify areas where future research could advance knowledge in the 
field. By focusing on programs that connect students and adults specifically on STEM content 
and careers, we can begin to dissect the possible benefits of such programs, both in relation to 
the mission of promoting interest in STEM fields, as well as supporting adolescents’ 
developmental well-being. 

Setting: 

This synthesis draws from published research about STEM experts in diverse settings. 

Population / Participants / Subjects: 

This synthesis project draws from published research about using STEM experts who 
work with K-12 students to engage them in STEM subjects, and presumably, improve their 
attitudes toward, and outcomes related to STEM subjects. 

Intervention / Program / Practice: 

This project is a synthesis of research evidence about multiple types of interventions and 
programs designed to improve K-12 students’ attitudes, persistence, and achievement in STEM 
subjects. 

Research Design: 

Our intent was to conduct a meta-analytic review, and to identify the specific practices 
for which sufficient credible causal research existed. Instead, our search identified numerous 
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papers describing programs, implementation, and general strategies for volunteerism in K-12 
education; surprisingly, the number of studies that reported on student outcomes was much 
smaller. We found papers whose studies addressed quite diverse programmatic goals, and were 
correspondingly diverse in the types of STEM experts, nature of their involvement, and 
centrality of their roles to the overall program goals. Most papers described program evaluations 
designed to provide information for program developers and operators, through rich descriptions 
of program implementation and outcomes. Further, the large majority of papers described 
programs for which STEM experts represented one of many program elements that varied in 
importance. The consequence of finding far fewer papers than expected led us to shift the focus 
away from synthesizing research results to make specific recommendations about a research- 
based agenda, and instead toward describing the research landscape more generally. 

Data Collection and Analysis: 

The literature search initially and purposefully included papers that described projects, 
programs, and initiatives that targeted students’ general STEM interest and engagement, as well 
as student retention and academic achievement in STEM courses, and students’ aspirations and 
plans for post-secondary careers and education. Initial searches surfaced thousands of articles 
about adjacent and related topics. Coders systematically screened bibliographic material (e.g., 
abstracts, keywords, citations, etc.) to group articles into “relevant” and “not relevant for our 
purposes” categories. Ultimately, we identified over 450 articles, reports, books, dissertations, 
and other sources, and were deemed potentially topically relevant for the literature review. 

The bibliographic information for sources identified as potentially relevant was entered into a 
screening database and systematically reviewed by trained coders who used a standardized 
screening protocol. The criteria for an initial review included: 

1. The focus of the paper or article was on a program (or programs) that engages K-12 students in 
STEM activities, in and/or outside of school (e.g. afterschool programs, camps, and competitions), 
using adults or older students as mentors or volunteers to increase student engagement, interest, 
persistence and achievement in STEM education (and ultimately) STEM careers; and 

2. The paper described specific programs, practices, activities etc., rather than more general discussions 
about (a) why student engagement in STEM is important, (b) guidance or recommendations for 
establishing programs to engage K-12 students, or (c) policies related to STEM engagement. 

Papers that did not meet these two criteria were screened out, resulting in 235 articles. The 
study team obtained the full text of each of those studies. The next procedural step included a 
more specific review than the initial screening process described above, by applying two 
additional criteria. Specifically, we focused on those papers that: 

1 . Described programs that used a wide range of adults or mentors who were employed by organizations 
engaged in STEM-related functions, whether technology, life sciences, analytics, engineering, or 
other STEM fields, or were engaged in academic pursuit in a STEM-related field (e.g., biology, 
medicine, IT, etc.) whether as faculty, postdoctoral researchers, graduate students, or undergraduates 
in a university STEM program. 

2. Explicitly described empirical research about student outcomes (i.e., reported data on student 
academic, attitudinal, or behavioral outcomes, and were described with sufficient detail for reviewers 
to characterize the nature of data collection activities). 


SREE Spring 2015 Conference Abstract Template 


4 



Articles that met the criteria were reviewed in depth, and information about those articles 
was entered into a database. Those not meeting our criteria were set aside. Ultimately, 29 articles 
were deemed eligible for inclusion. The selected articles were then reviewed by trained coders 
who recorded and summarized key pieces of information into a database using a structured 
coding protocol. Information entered into the database about each article could be (and was) 
further coded, and later synthesized. The coding process was iterative; papers were re-examined 
in light of new themes and information that emerged from the review and synthesis processes. 

Findings / Results: 

The 29 papers included in the final review were published or posted between 1986 and 2013. 
Most were published in 2000 or later (Appendix 2). The studies were disseminated via peer- 
reviewed journals, government reports, conference papers, dissertations, and program websites. 
Some were unpublished reports posted on program or project websites. All of the papers 
described studies that in some way measured the impact or effectiveness of programs in which 
STEM professional or other adults work directly with K-12 students. 

Conclusions: 

Several themes emerged from the literature search and from the research findings of the 29 
papers in the review. First, the search generated numerous articles on STEM experts working 
with K-12 students. The abundance of research addressing this topic suggests that use of STEM 
experts, in some fashion, is quite prevalent, and that the inclusion of STEM professionals in 
STEM activities with K-12 students is a widely implemented educational approach. In addition, 
most of the articles describe more formative than summative research on programs using STEM 
experts, and further, most describe and explore aspects of program implementation. Of the large 
number of studies initially identified, only 29 articles involved comparative research designs, 
either through the inclusion of a comparison group or a pre- to post-program comparison. 

Another theme is that the focus of research varied widely in terms of the levels of detail 
provided on STEM professionals’ roles and importance in program activities. Some articles 
highlight the roles played by STEM exerts, and explicitly identify them as a key part of the 
educational approach of the programs studied. Other articles mention STEM expert involvement 
in program activities in passing, without describing the nature of adults’ involvement, how the 
experts fit into the larger educational approaches used by the programs, or how the experts are 
recruited, trained, supported, or reviewed. Consequently, it is difficult to ascertain whether the 
variation in STEM roles across articles reflects the variation in the importance of their actual 
roles in programs, or whether it is an artifact of different reporting styles and foci. The variation 
in details contained in reports and in programs themselves makes it difficult to draw conclusions 
about adults’ contributions across programs. 

Additionally, the large majority of studies were not methodologically rigorous efficacy or 
effectiveness studies. Few used designs that can support casual conclusions about the impacts of 
the programs on student outcomes, chiefly because they employed quasi-experimental and pre- 
post one group designs. As noted above, these types of designs cannot provide conclusive 
evidence that programs “cause” positive student outcomes. Similarly, quasi-experimental and 
pre-post designs cannot sufficiently rule out the possibility that differences observed between 
students at the beginning and end of a program, or between program participants and 
nonparticipants, are attributable to factors outside of the program itself, such as differences in 
schools or characteristics of students prior to program attendance. 
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